Building Bing Developer Assistant MSR-TR-2015-36

Software developers heavily rely on code snippets and API usage examples searched on the Internet. This paper presents Bing Code Search, a Visual Studio extension that allows developers to write, within an IDE, free-form natural language questions, and get C# code snippets answering those questions. Bing Code Search automatically adapts the suggested snippets into the user’s programming context via variable renaming, and records users’ interactions to improve its suggestions. Compared to prior related research, Bing Code Search provides a complete automation of the full search-paste-adapt process. Three weeks after we released this free extension, more than 20,000 users downloaded it; and they issue on average 3,000 queries per day. We believe that Bing Code Search is the most widely used tool in its category. In the following, we fully describe our framework, and draw clear empirical evidence of the benefits of Bing Code Search: (1) From our evaluation benchmark, compared with Bing’s result, Bing Code Search delivers more relevant snippet solutions. (2) In a controlled experiment, it was able to save developers 28% of time on completing API related tasks. (3) Telemetries collected from thousands of users show some users already built up the habit of using the tool: they issue multiple queries to solve a complex task, or use it as a fast auto-completion.